Papers with Statistical significance testing
NLPStatTest: A Toolkit for Comparing NLP System Performance (2020.aacl-demo)
Copied to clipboard
| Challenge: | Statistical significance testing is used to compare NLP system performance, but p-values alone are insufficient because statistical significance differs from practical significance. |
| Approach: | They propose a three-stage procedure for comparing NLP system performance and a toolkit that automates the process. |
| Outcome: | The proposed procedure is based on a three-stage procedure and compares it with existing statistical testing toolkits. |
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing (P18-1)
Copied to clipboard
| Challenge: | Statistical significance testing is a standard statistical tool designed to ensure that experimental results are not coincidental. |
| Approach: | They propose a protocol for statistical significance test selection in NLP setups . they propose he proposes a survey of the most relevant tests to help guide the protocol . |
| Outcome: | The proposed protocol includes a survey of the most relevant tests. |
Faithful Model Evaluation for Model-Based Metrics (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies do not consider variance change due to metric model errors, which can lead to wrong conclusions. |
| Approach: | They establish the mathematical foundation of significance testing for model-based metrics . they show that metric errors can change the conclusions in certain experiments . |
| Outcome: | The proposed method can be used to derive accurate conclusions using model evaluations. |
Context Minimization for Resource-Constrained Text Classification: Optimizing Performance-Efficiency Trade-offs through Linguistic Features (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models have transformed text classification, but their computational demands often render them impractical for resource-constrained settings. |
| Approach: | They propose a linguistically-grounded framework for context minimization that leverages theme-rheme structure to preserve critical classification signals while reducing input complexity. |
| Outcome: | The proposed framework preserves critical classification signals while reducing input complexity. |